The means by which users interact with computer systems and electronic devices has evolved throughout the years. Prior to the graphical user interface (GUI), the keyboard was the primary means for interacting with a computer system. In addition to typing and editing text, the keyboard was used to control other aspects of the computer system by way of a command line interface (CLI).
Through the years, the keyboard changed very little as the computer systems evolved from a CLI to a GUI. Some attempts were made to adapt the keyboard to the GUI environment. For example, keystroke “shortcuts” were introduced to directly invoke various output actions without the need to navigate the spatial selection paradigm of the GUI, whereby navigating the spatial selection paradigm involves mapping movements that are detected using some physical device (e.g., a mouse) to the virtual interface of the GUI in order to interact with the user interface (UI) elements of the GUI. However, these shortcuts were and remain difficult to remember and inconsistent across different applications and platforms. There is often no on-screen indication to remind the user of the shortcuts. Moreover, the “flag” keys (e.g., alt, ctrl, command, etc.) used to invoke these shortcuts are located in non-standardized peripheral positions (e.g., bottom left corner) on different keyboards. Therefore to invoke a keyboard shortcut, the user often performs awkward hand movements and finger extensions or removes one hand from the keyboard hand position in order to reach a flag key that is part of some keyboard shortcut. Further hindrance to the use of keyboard shortcuts is the limited number of flag keys that limits the number of possible keyboard shortcuts and the customizability of these shortcuts.
Consequently, the mouse evolved to become the primary means to navigate the GUI and to invoke output actions. Using the mouse, users can indirectly invoke output actions by moving a mouse pointer to and clicking on various selectable control areas or UI elements they perceive on-screen. Such various selectable control areas or UI elements include icons, menu items, graphical sliders, hyperlinks, and other such interactive graphical elements that are clickable when displayed on-screen. When clicked, each UI element performs an output action such as launching an application, opening a file (e.g., documents, images, videos, audio files, etc.), accessing folders, adjusting a widget, executing scripts, issuing calls to one or more application programming interfaces (APIs) or libraries, and performing other OS, GUI, or application operations.
The current technological shift is moving users towards touchscreen interfaces. Touchscreen interfaces allow users to directly invoke output actions through the UI elements they perceive on-screen. Instead of moving a mouse pointer, users can move their hand and fingers to the location of various UI elements to invoke corresponding output actions.
While intuitive, the mouse and the touchscreen interfaces have various shortcomings that restrict their use. Firstly, the speed by which a user can navigate the GUI and invoke output actions using the mouse or the touchscreen interface is mitigated by the spatial dependence of these input devices. Specifically, users must visually identify a particular UI element before they are able to select and invoke an output action that is associated with the particular UI element. With the mouse, the user identifies a current location of the mouse pointer before identifying a path to the particular UI element. With the touchscreen interface, the user identifies the on-screen location of the particular UI element before moving their hand and fingers to that element, and then invokes an output action that is associated with the particular UI element with a touch gesture.
Subsequent invocation of the same output action requires the same spatial identification steps. Each time the user wants to invoke that output action, the user first has to identify the location of the particular UI element on-screen and then make precise spatially relative movements. As a result, neither the mouse nor the touchscreen interface can leverage muscle memory for repeated invocation of output actions since the actual physical motion is different every time. Without the use of muscle memory, users are unable to quickly and subconsciously recreate a set of operations that invoke output actions. When invoking output actions that are used once or very infrequently (such as clicking on a link in a web browser), memorizing commands to invoke output actions is neither desirable nor beneficial. However, the more an output action is utilized, the more benefit there is in allowing the user to develop and leverage muscle memory so that the tedious and slow spatial identification and precise movement steps can be eliminated.
Secondly, access to a desired output action may be obscured by the implied three dimensional nature and limited screen size of the GUI. For example, a particular UI element that is used to invoke a desired output action may be obscured by one or more application windows, or may be obscured by being located within various menus, graphical ribbons, or folders. Therefore, to access the desired output action, the user has to memorize and navigate through the GUI (e.g., windows, menus, ribbons, folders, etc.) before scanning again to locate the desired output action. Some UI elements such as hierarchical menus require several iterations of this action/disclose/scan cycle. The mouse and touchscreen interface, therefore, often do not provide a direct means to invoke an output action.
A desktop icon may be used as a “shortcut” to invoke an output action (such as opening a document). However, the user still has to identify and move the mouse or hand to the location of the icon before clicking the icon. Additionally, the desktop may quickly be overrun with icons, making identification difficult. Also, the implied three dimensional nature of the GUI may obscure desktop icons with various windows.
Thirdly, mouse and touchscreen interfaces are not well suited for inserting text (typing). As a result, these input devices are often used in conjunction with a physical keyboard. However, alternating between a keyboard and mouse/touchscreen introduces a second layer of spatial recognition. In order to switch to a different input device, the user identifies the physical location of the different input device before switching to that different input device. The physical movement of the hand from the mouse/touchscreen to the keyboard and vice versa introduces delay and creates discontinuity in the use of the computer system. This delay is further exacerbated when moving away from or back to the keyboard as the user loses and must reorient their hand to the precise home position on the keyboard.
Fourthly, the mouse and the touchscreen interfaces are continuous input devices that require fine motor skills. A touchscreen gesture or a mouse click will not invoke the output action for a UI element unless the touchscreen gesture or mouse click/drag occurs within a specifically defined region of the UI element, which may be quite small. In contrast, a key-based input device (e.g., keyboard) provides discrete inputs to the computer system. More specifically, each key of a key-based input device that is pressed provides a discrete input (e.g., entry of an alphanumeric character) to the computer system.
Accordingly, there is a need for further UI evolution. There is a need to enhance the UI to reduce the time, energy, and frustration associated with using a spatial input device in order to invoke frequently used output actions. To improve the efficiency with which the user invokes an output action, there is a need to incorporate muscle memory into the UI. There is also a need to better unify the invocation of output actions with character insertion (typing) so that a single input device can be used as the primary means to invoke output actions as well as inserting characters and thereby minimize or eliminate hand movement. There is a therefore need to provide additional expressivity through a key-based input device so that the discrete inputs of the device can be used both to perform character insertion and invoke a wide variety of output actions without the need to perform constant spatial identification in the GUI.